PySpark - RDD: Hands-on Tracing Quiz

This workbook guides you through tracing partition-level data transformations and coalescing boundaries using PySpark RDDs.

1. The Dataset & Cluster Config

Assume a click logs raw dataset:

device_1,2026-05-26T12:00:00,SUCCESS
device_2,2026-05-26T12:01:00,FAIL
device_1,2026-05-26T12:02:00,SUCCESS
device_3,2026-05-26T12:03:00,SUCCESS
device_2,2026-05-26T12:04:00,SUCCESS
device_1,2026-05-26T12:05:00,FAIL

Partition Allocations (4 Initial Partitions)

• Partition 0: Rows 1-2
• Partition 1: Row 3
• Partition 2: Rows 4-5
• Partition 3: Row 6

2. Tasks

Task 1: Trace mapPartitions Count

Write a PySpark pipeline using .mapPartitions() to count how many records reside in each partition before performing any filters. Show the exact output format returned to the driver.

Task 2: Filter and Tuple Map Tracing

Write code to filter for SUCCESS logs and map each to (device_id, 1). Trace the elements inside each of the 4 partitions.

Task 3: Coalesce Down-Partitioning

We execute .coalesce(2) to down-partition the dataset. Trace how Spark merges partitions without a shuffle and detail the final partition structures.

3. Step-by-Step Solutions

Solution 1: mapPartitions Count

• PySpark Code:

raw_rdd = sc.textFile("telemetry.txt", minPartitions=4)

def count_elements(records_iterator):
    count = sum(1 for _ in records_iterator)
    yield count

partition_counts = raw_rdd.mapPartitions(count_elements).collect()
print(partition_counts)

• Tracing Matrix:
  • Partition 0: [device_1, SUCCESS, device_2, FAIL] Yields 2
  • Partition 1: [device_1, SUCCESS] Yields 1
  • Partition 2: [device_3, SUCCESS, device_2, SUCCESS] Yields 2
  • Partition 3: [device_1, FAIL] Yields 1
• Result: [2, 1, 2, 1]

Solution 2: Filter & Map Tracing

• PySpark Code:

success_rdd = raw_rdd.filter(lambda line: "SUCCESS" in line) \
                      .map(lambda line: (line.split(",")[0], 1))

• Partition Trace:
  • Partition 0: ("device_1", 1) (Row 2 failed and was dropped)
  • Partition 1: ("device_1", 1)
  • Partition 2: ("device_3", 1), ("device_2", 1)
  • Partition 3: Empty (Row 6 failed and was dropped)

Solution 3: Coalesce Down-Partitioning

• Coalesce Mechanics: .coalesce(2) merges adjacent partitions on the same executor/node without forcing a full network shuffle (wide dependency).
• Partition Merging:
  • New Partition 0 = Partition 0 (1 record) + Partition 1 (1 record) [("device_1", 1), ("device_1", 1)]
  • New Partition 1 = Partition 2 (2 records) + Partition 3 (0 records) [("device_3", 1), ("device_2", 1)]
• No data is shuffled across the network; it simply bundles the partitions locally, keeping execution narrow and extremely fast.